Skip to content

docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7

Jump to bottom
Open
Jarrod S Doucette (jdoucett) wants to merge 5 commits into
base: main
Choose a base branch
from
Open

docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) #7

Jarrod S Doucette (jdoucett) wants to merge 5 commits into from
+1,344 −1

Conversation

@jdoucett
Copy link
Contributor

@jdoucett Jarrod S Doucette (jdoucett) commented Jun 26, 2026 •
edited
Loading

Refreshed: the staged-hybrid RFC (review gate for Phase 2)

This rewrites docs/RFC_incremental_v2.md from the original leveled-only RFC into the
umbrella staged-hybrid design — matching what actually shipped (Phase 1) and what is
being built next (Phase 2). This PR is a review gate: no engine code lands until the
Phase-2 design here is approved.

What changed vs the old RFC

Phase 2 in one screen

Per target, the post-exclusion survivors are partitioned into independent Fortress
objects, each shipped through the unchanged make_zip_files → send_to_fortress
verify chain:

  • Solo — one file ≥ T_small (256 MiB, floor 100 MB); its own object; skips if unchanged.
  • Shard — files < T_small bucketed by k = hash(relpath) mod K (K frozen at baseline); whole-shard re-ship on any member change. No repack cascade.
  • The append win: a growing target's new big files become new solos; every existing object skips — zero re-read of already-archived bytes. Skip-unchanged also doubles as resume (a half-shipped object simply isn't in the manifest yet).
  • Per-target AGGREGATE reclaim (verify_target) — REQUIRED in the same slice as the first routed ship, or every routed target perpetually false-DRIFTs (reclaim dedups to the newest log = one object). Sums all objects; requires every object tar present; INV-E exclusion re-gate on the aggregate.
  • Reship-monitor — [RESHIP-WATCH] email when shard reship ratio > ~25%: the uniform-medium-churn signature and the only trigger to activate Phase 3.
  • Full restore (§2.9) is the capstone slice — object-model reassembly (disjoint union, so simpler than Phase-3 leveled restore: no overlay/deletion replay), completeness gate + per-object MD5 verify + mandatory reconciliation, --slot selective restore. It's what makes aggregate --delete trustworthy.
  • Local-mode only — refuses --globus (routing + manifest are on the launching host).
  • Manifest per target at {log_dir}/_vault/{badge}.manifest.json (dodges reclaim's non-recursive glob).

Design driver (settled, do not re-litigate)

The 2026-07-08 source_files measurement (809 logs, Negishi): standoff imagery is
stable (Phase-1 exclusion covers it); the data modalities GROW by appending
(modified≈0) — so append is the dominant Phase-2 win, which content-addressed
shards + solos deliver for free. Corrects the earlier du-based "omics re-analyzes
everything" read.

Build order (after approval) — same cadence as exclusion

  1. Routing primitives (behavior-neutral) + only_arcnames on make_zip_files.
  2. Archive wiring (the append engine) + config keys + --globus refusal + --fresh re-baseline.
  3. Aggregate reclaim (verify_target, required with slice 2's first routed ship).
  4. Reship-monitor.
  5. Full restore (restore.py) — the capstone.

Tests ship with every slice; python3 -m unittest discover -s tests before merging.

🤖 Generated with Claude Code

@jdoucett Jarrod S Doucette (jdoucett) mentioned this pull request Jun 29, 2026
Merged
Jarrod S Doucette (jdoucett) added a commit that referenced this pull request Jul 8, 2026
@jdoucett
docs: refresh RFC #7 into the staged-hybrid design (Phase 2 = size-ro…
34ad875 
…uting/append)

Rewrites docs/RFC_incremental_v2.md from the leveled-only RFC into the umbrella
staged-hybrid RFC that matches what actually shipped and what is being built next:

- Phase 1 (exclusion): DONE/merged; recap + pointer to docs/EXCLUSION_SPEC.md.
- Phase 2 (size-routing/APPEND): the new build target, full design — solo objects
  >= T_small (256 MiB, floor 100 MB), content-addressed shards (hash(relpath) mod K,
  K frozen at baseline), skip-unchanged (the append win, doubles as resume),
  per-target AGGREGATE reclaim (verify_target — REQUIRED same slice or every routed
  target false-DRIFTs), reship-monitor (the Phase-3 trigger), local-mode only
  (refuses --globus). 4 reviewable slices with tests.
- Phase 3 (leveled L0/L1/L2): DEFERRED-DORMANT banner; the prior leveled RFC content
  (Path A/B, restore, chain-aware reclaim, Negishi spike) preserved intact as the
  Phase-3 reference.
- Appendix: the PR #8 primitives (badge_of/make_blonde/enumerate_source_catalog/
  floor_s) reused across phases, with current line refs.

Design driver = the 2026-07-08 source_files measurement: standoff imagery is stable
(exclusion covers it); the data modalities GROW by appending -> append is the
dominant Phase-2 win. Review gate before any engine code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jdoucett Jarrod S Doucette (jdoucett) changed the title docs: RFC — in-engine leveled incremental backup (v2) docs: RFC — staged-hybrid incremental backup (Phase 2 = size-routing/append) Jul 8, 2026
@jdoucett
docs: refresh RFC #7 into the staged-hybrid design (Phase 2 = size-ro…
b617c58 
…uting/append)

Rewrites docs/RFC_incremental_v2.md from the leveled-only RFC into the umbrella
staged-hybrid RFC that matches what actually shipped and what is being built next:

- Phase 1 (exclusion): DONE/merged; recap + pointer to docs/EXCLUSION_SPEC.md.
- Phase 2 (size-routing/APPEND): the new build target, full design — solo objects
  >= T_small (256 MiB, floor 100 MB), content-addressed shards (hash(relpath) mod K,
  K frozen at baseline), skip-unchanged (the append win, doubles as resume),
  per-target AGGREGATE reclaim (verify_target — REQUIRED same slice or every routed
  target false-DRIFTs), reship-monitor (the Phase-3 trigger), local-mode only
  (refuses --globus). 4 reviewable slices with tests.
- Phase 3 (leveled L0/L1/L2): DEFERRED-DORMANT banner; the prior leveled RFC content
  (Path A/B, restore, chain-aware reclaim, Negishi spike) preserved intact as the
  Phase-3 reference.
- Appendix: the PR #8 primitives (badge_of/make_blonde/enumerate_source_catalog/
  floor_s) reused across phases, with current line refs.

Design driver = the 2026-07-08 source_files measurement: standoff imagery is stable
(exclusion covers it); the data modalities GROW by appending -> append is the
dominant Phase-2 win. Review gate before any engine code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jdoucett
docs(rfc): make full restore the capstone slice of Phase 2 (not a Pha…
febf616 
…se-3 deferral)

Full restore now §2.9 (object-model reassembly: disjoint union, so simpler than the
deferred Phase-3 leveled restore — no overlay/deletion replay/chain fold) and slice 5
of the build order (§2.10). --slot selective restore subsumes single-file restore.
Restore is what makes reclaim's aggregate --delete trustworthy (prove round-trip first).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jdoucett
docs(rfc): sync fingerprint/slot detail with slice-1 impl; mark slice…
55c14c6 
… 1 DONE

- shard fingerprint = {members, byte_sum, digest} (shake_128 over sorted
  (arcname, floor_s(mtime), size) triples) — closes the same-count/same-bytes
  reshuffle hole the plain mtime_sum draft had.
- solo slot = 64-bit hex(shake_128(arcname)[:8]) (was b32[:4]); route() guards
  the residual collision.
- §2.10 slice 1 marked [DONE] with the real helper names + test coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jdoucett
feat(routing): Phase-2 size-routing primitives (slice 1, behavior-neu…
dadb1d4 
…tral)

Adds the pure, additive size-routing helpers to archive.py — nothing wires them
into the archive or reclaim flow yet (behavior-neutral; full suite 136 -> 172 green):

- route(catalog, t_small, shard_count) -> [{slot, kind, arcnames}] partition:
  files >= T_small become solo objects; smaller files bucket into content-addressed
  shards k = shake_128(arcname) mod K (no repack cascade -> cheap APPEND).
- shard_of / solo_slot / shard_slot / derive_shard_count (K = ceil(small_bytes /
  shard_target), frozen per badge at baseline).
- object_fingerprint (solo {mtime_s,size}; shard {members,byte_sum,digest}) +
  object_unchanged (the skip/append signal, also the resume mechanism).
- clamp_t_small (100 MB floor), budget_warnings (warn-only max_solo/max_objects/
  oversize-shard), vault_dir_for/manifest_path/load_manifest/write_manifest
  (atomic tmp+os.replace, best-effort; _vault child dodges reclaim's glob).
- make_zip_files gains an optional only_arcnames filter (no caller yet) so a later
  slice can ship exactly one routed object; keys stay arcname-relpath (invariant #2).

Tests: tests/test_routing_primitives.py (36 cases) — boundary partition, shard
stability, K frozen, fingerprint change detection incl. reshuffle, skip vs
shard-append, only_arcnames subset with matching checksum keys, manifest round-trip.

See docs/RFC_incremental_v2.md §2 (slice 1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jdoucett Jarrod S Doucette (jdoucett) mentioned this pull request Jul 8, 2026
Open
@jdoucett
docs(rfc): manifest must be shared+durable (Depot log_dir), not home/…
8cb9232 
…scratch

Jarrod review calls:
- K = derived-and-frozen (confirmed; already in derive_shard_count).
- Manifest location: it is durable cross-run state reclaim must find, so it MUST
  live on shared+durable storage (the deployment Depot logs dir that log_dir already
  points at), NEVER home (the current default trap) or scratch (purged -> reclaim
  can't reconstruct the aggregate -> every routed target false-DRIFTs). Add a slice-2
  durability guard in load_config; tmp_dir (staging zips) stays in scratch.
- --globus: reframed. Real invariant = manifest on a shared log_dir readable by both
  the run and reclaim, which Negishi (the only launch host) satisfies either way; the
  split only happens off-box, which we don't do. Globus is off on Negishi now, so
  size_routing stays local-only as a cheap guard (untested stateful path), not a deep
  limitation — allowable under --globus later with the durability guard in place.

§2.4 manifest-location invariant + guard; §2.8 --globus reframe; §2.10 slice-2 guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in to join this conversation on GitHub.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jdoucett